Method and system for detecting change to structure by using drone

ABSTRACT

Disclosed herein is a method for an image analysis server to detect a change to a structure by using a drone. The method for an image analysis server to detect a change to a structure by using a drone includes: receiving images of a specific inspection target structure taken at different time points by a drone; detecting the difference between an image taken at a first time point and an image taken at a second time point based on the received images; and detecting a change to the inspection target structure via the detected difference, and generating a risk signal and then transmitting it to an administrator terminal.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.17/687,579 field Mar. 4, 2022, which claims under 35 U.S.C. § 119(a) thepriority benefit of Korean Patent Application No. 10-2021-0045135 filedon Apr. 7, 2021, the disclosures of all application of which are herebyincorporated by reference.

BACKGROUND 1. Technical Field

The present invention relates generally to a method and system fordetecting a change to a structure by using a drone, and moreparticularly to a method and system for detecting a change to astructure by analyzing images, taken via a drone at different timepoints, based on a machine learning algorithm.

2. Description of the Related Art

Concrete bridges, high-rise buildings, dams, etc. are easily influencedby external environments after design and completion. Various types ofdamage resulting from unknown causes may occur under the influence ofsuch external environments.

Currently, administrators regularly inspect individual structures, ordetect damage to structures by using well-known non-destructiveinspection methods.

For example, an inspector photographs portions deemed necessary forinspection and also inspects them with the naked eye while boarding aboarding device installed at a high location.

However, according to this method, problems arise in that there is therisk of a safety-related accident and the range in which photographingcan be performed is limited.

Furthermore, changes to structures are made so fine that it is difficultto identify them with the naked eye. There are cases where a change to astructure cannot be easily identified even when an inspector takesimages while taking the risk of a safety-related accident.

Therefore, there is a demand for technology capable of accuratelydetecting a change to a structure attributable to an externalenvironment and detecting a change to a structure in a convenient waywithout the risk of a safety-related accident.

SUMMARY

An object of the present invention is to overcome the problems of theconventional technologies described above.

An object of the present invention is to accurately detect a change,such as damage, to an inspection target structure in a convenient wayusing a drone.

Another object of the present invention is to, via a machine learningalgorithm, acquire feature values from images taken at different timepoints and detect a change to a structure through the comparativeanalysis thereof, thereby enabling accurate analysis related to damageto the structure.

The objects of the present invention are not limited to the objectsdescribed above, and other objects not described above will be clearlyunderstood from the following description.

According to an aspect of the present invention, there is provided amethod for an image analysis server to detect a change to a structure byusing a drone, the method including: receiving images of a specificinspection target structure taken at different time points by a drone;detecting the difference between an image taken at a first time pointand an image taken at a second time point based on the received images;and detecting a change to the inspection target structure via thedetected difference, and generating a risk signal and then transmittingit to an administrator terminal.

Detecting the difference may include: acquiring feature maps forrespective images by learning an image taken first for the specificinspection target structure and images taken at different time pointsthereafter by using a machine learning algorithm; and acquiring thedifference between the feature map acquired based on the image takenfirst and the feature map acquired based on each of the images takensubsequently by using a Euclidean distance analysis method.

The method may further include predicting the change pattern of thespecific inspection target structure in the future based on a pluralityof images of the specific inspection target structure taken at differenttime points.

Generating the risk signal and then transmitting it may includegenerating a risk signal when the area of a detected change area or thearea value of a change per unit time is equal to or larger than a presetvalue.

According to another aspect of the present invention, there is provideda system for detecting a change to a structure by using a drone, thesystem including: an image acquisition unit configured to receive imagesof a specific inspection target structure taken at different time pointsby a drone; an image learning unit configured to detect the differencebetween an image taken at a first time point and an image taken at asecond time point based on the received images; and a change detectionunit configured to detect a change to the inspection target structurevia the detected difference, and generating a risk signal and thentransmitting it to an administrator terminal.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentinvention will be more clearly understood from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a view schematically showing the configuration of a system fordetecting a change to a structure by using a drone according to anembodiment of the present invention;

FIG. 2 is a block diagram illustrating in detail the operation andconfiguration of an image analysis server according to an embodiment ofthe present invention;

FIG. 3 is a diagram showing an example of a machine learning algorithmused for image analysis according to an embodiment;

FIG. 4 is a view showing a change map acquired for a specific inspectiontarget structure in the period from January to April according to anembodiment of the present invention; and

FIG. 5 is a view showing a result image generated by performing changeprediction according to an embodiment of the present invention.

DETAILED DESCRIPTION

For the following detailed description of the present invention,reference is made to the accompanying drawings that show by way ofillustration specific embodiments via which the present invention may bepracticed. These embodiments will be described in sufficient detail toenable those skilled in the art to practice the present invention. Itshould be understood that various embodiments of the present inventionare different but are not necessarily mutually exclusive. For example, aspecific shape, structure, and/or feature described herein may beimplemented as another embodiment without departing from the spirit andscope of the invention with respect to one embodiment. In addition, itshould be understood that the locations or arrangement of individualcomponents within each disclosed embodiment may be changed withoutdeparting from the spirit and scope of the present invention.Accordingly, the following detailed description is not intended to betaken in a limiting sense, and the scope of the present invention,together with all ranges equivalent to the appended claims ifappropriately described, is limited only by the appended claims. Likereference numerals in the drawings refer to the same or similarfunctions throughout various aspects.

The embodiments of the present invention will be described in detailbelow with reference to the accompanying drawings so that those ofordinary skill in the art to which the present invention pertains caneasily practice the present invention.

FIG. 1 is a view schematically showing the configuration of a system fordetecting a change to a structure by using a drone according to anembodiment of the present invention.

Referring to FIG. 1 , the system for detecting a change to a structureby using a drone according to the present embodiment may include a drone100, an image analysis server 200, and an administrator terminal 300.

The drone 100, the image analysis server 200, and the administratorterminal 300 may communicate over an intercommunication network, e.g., aLoRa communication network, a mobile communication network, a local areanetwork (LAN), a metropolitan area network (MAN), a wide area network(WAN), the World Wide Web (WWW), and/or a wireless fidelity (Wi-Fi)communication network.

The drone 100 may include a flight control module, a Global PositioningSystem (GPS) module, a photographing module, a level maintenance module,a driving module, a wireless communication module, and a battery module.The drone 100 receives a command input via a controller operated by adriver through wireless communication, and then flies.

The flight operation control and photographing operation control of thedrone 100 may be performed by the controller.

For example, the drone 100 is flown and moved to a location desired bythe driver according to a command input via the controller operated bythe driver, and performs the operation of photographing an inspectiontarget structure ST.

According to another embodiment, the drone 100 may fly in an autonomousflight mode and move to a location close to the inspection targetstructure ST. For example, the drone 100 may receive data (e.g.,information about the location of a destination, or the like) input viaa controller or the like over a wireless communication network, and maythen generate a path for autonomous flight. Furthermore, the drone 100may set up a flight plan adapted to fly along the generated path, andmay generate a signal adapted to control movement in accordance with theflight plan. The generated signal is provided to the driving module andused to fly the drone.

The drone 100 may further include a proximity sensor, and may furtherinclude a collision avoidance alarm system. Furthermore, the drone 100may further include a system for returning the drone 100 to anappropriate flight path when the drone 100 deviates from the generatedflight path.

The drone 100 may perform the operation of switching from the autonomousflight mode to the manual flight mode or the operation of switching fromthe manual flight mode to the autonomous flight mode. For example, thedrone 100 may switch the flight mode thereof from the autonomous flightmode to the manual flight mode or from the manual flight mode to theautonomous flight mode based on a signal received from an interfacedevice provided in the controller operated by the driver.

The photographing operation is performed by the photographing modulemounted in the drone 100, and a taken image or video is transmitted tothe image analysis server 200.

The image analysis server 200 receives an image or video taken andacquired by the drone 100 and analyzes it. The image analysis server 200according to an embodiment detects a change to a specific inspectiontarget structure ST via a machine learning algorithm based on images ofthe corresponding inspection target structure ST taken at different timepoints. Furthermore, when it is determined that the detected change isin a risky state, this information is transmitted to the administratorterminal 300.

The administrator terminal 300 serves to transmit a command adapted tocontrol the analysis operation of the image analysis server 200, and toreceive the results of image analysis acquired by the image analysisserver 200. The administrator terminal 300 has a computation function,and may be implemented in any form as long as it is a device capable ofcommunicating with the outside. For example, the administrator terminal300 may be implemented as a smartphone, a tablet personal computer (PC),a desktop, a laptop, a notebook, a personal digital assistant (PDA), orthe like, but is not limited thereto.

FIG. 2 is a block diagram illustrating in detail the operation andconfiguration of an image analysis server according to an embodiment ofthe present invention.

Referring to FIG. 2 , the image analysis server 200 may include an imageacquisition unit 210, an image learning unit 220, a change detectionunit 230, and a change prediction unit 240.

The image acquisition unit 210, the image learning unit 220, the changedetection unit 230, and the change prediction unit 240 may be programmodules or hardware capable of communicating with external devices. Theprogram modules or hardware may be included in the image analysis server200 in the form of an operating system, application program modules, orother program modules, and may physically be stored in various types ofknown storage devices. Meanwhile, these program modules or hardwareinclude, but are not limited to, routines, subroutines, programs,objects, components, and/or data structures that perform specific tasksto be described later or execute specific abstract data types accordingto the present invention.

The image acquisition unit 210 serves to receive a still or moving imagetaken by the drone 100. An image taken immediately after the completionof an inspection target structure or an image taken first becomes areference image of the inspection target structure. Furthermore, one ormore images taken at different time points by the drone 100 are alsoacquired. In order to identify images of the same inspection targetstructure taken a plurality of times, when an image is received from thedrone 100, information about a photographing location, a photographingdate, photographing time, and/or the like may also be received asmetadata.

The image learning unit 220 performs learning via a machine learningalgorithm based on images acquired by the image acquisition unit 210.

FIG. 3 is a diagram showing an example of a machine learning algorithmused for image analysis according to an embodiment.

Referring to FIG. 3 , machine learning is performed via a referenceimage of a specific inspection target structure first acquired at timet0 and a dataset of images acquired at times t1, t2, . . . , tn.

The present invention adopts a learning method using a deep learningtechnique, which is a type of machine learning. Machine learning is abranch of artificial intelligence, and has evolved from studies ofpattern recognition and computer learning theories.

Machine learning improves a knowledge base by using surroundingenvironments as training elements. A specific task is performed usingthe improved knowledge base, and the information obtained during theperformance of the task is reflected in the training elements again.Machine learning is a technique that studies and constructs a system andalgorithm for performing learning based on empirical data in the abovemanner, making predictions, and improving its own performance. Machinelearning algorithms use a method of constructing a specific model tomake a prediction or decision based on input data

Machine learning may be classified into a method of rote learning anddirect provision of new knowledge, a supervised learning method, alearning-by-analogy method, and an inductive learning method accordingto their learning strategies. The present invention may use at least oneof the above learning methods.

The reason why the performance of the deep learning model was improvedis that it was possible to train a large-scale model with large-scaledata. A major contribution to this was made by a convolutional neuralnetwork that is adopted by the present invention.

A convolutional neural network uses convolution kernels to automaticallyextract useful features and representations from high-dimensional datasuch as images. Through this, the same parameter value is obtained evenwhen locations are different, and the number of dimensions to be learnedmay be reduced by reducing the number of parameters. Excessive learningmay be prevented and useful features may also be extracted by using aconvolutional neural network model.

A convolutional neural network is basically designed to solve supervisedlearning problems, and is focused on discriminative learning to classifyinput data. This may improve the performance of pattern classification,and may construct complex features and representations by itself using aconsiderably large number of neuron layers. A convolutional neuralnetwork is a neural network that improves performance by adding aconvolution layer and a sub-sampling layer to the fully connected layerof the hidden layer of a conventional neural network and also performingstructural subdivision. Deep learning using such a convolutional neuralnetwork exhibits considerably excellent accuracy compared to othertechniques.

A convolutional neural network performs feature extraction viaconvolution and performs classification via a neural network. In imageprocessing, convolution refers to image processing performed using amask having weights. Convolution is a process of putting a mask on aninput image, multiplying the pixel values of the input image by theweights of the mask, and then determining the sums to be the pixelvalues of the output image. A mask used for image processing is referredto as a filter, a window, or a kernel.

The convolution values are calculated while moving the mask afterputting the mask on the input image. The purpose of this is to extractfeatures from an image. When features are extracted, robust featuresthat adapt well to environments such as image distortion or deformationare obtained by performing extraction on multiple images.

When convolution and sub sampling are repeated, extracted featuresremain. When results are input to each input terminal of a neuralnetwork, learning is performed.

A convolutional neural network used in deep learning is an artificialneural network that understands images and performs various types ofimage processing, such as the extraction of high-level abstractedinformation, e.g., feature values, or the generation of images havingnew textures, on the images, and is being studied in the field ofcomputer vision.

In an embodiment of the present invention, a feature map for a referenceimage and a feature map for each of a plurality of images taken atdifferent time points may be acquired by learning a plurality of imagesthrough such a convolutional neural network.

Referring back to FIG. 2 , the change detection unit 230 analyzes thedifferences between the feature map acquired through the reference imageand the feature maps acquired through the images taken at differenttimes based on the feature maps acquired by the image learning unit 220.

For this analysis, a Euclidean distance analysis method may be utilized.The differences between the acquired reference and other feature mapsmay be determined on a pixel basis via the Euclidean distance analysismethod. A change to an inspection target structure between time t0 atwhich the first image was taken and the later time point may bedetermined by detecting the intra-image range in which a differencevalue is equal to or larger than a threshold value.

As results of the analysis, a “reference image map” acquired through thereference image and a “change image map” acquired through an image takenthereafter may be acquired.

The feature values included in the change image map may function as amask in subsequent image analysis. A mask for detecting a change to aninspection target structure may be periodically updated by repeatinglearning and change detection through images taken at different timepoints.

The time point at which the structure was changed from the first time tothe second time may be clearly identified by using a mask, acquired bylearning the image acquired at the first time, to analyze the imageacquired at the second time.

FIG. 4 is a view showing a change map acquired for a specific inspectiontarget structure in the period from January to April according to anembodiment of the present invention.

The change detection unit 230 may determine the risk of a structurethrough such a change map.

According to an embodiment, the change detection unit 230 may determinewhether there is a risk based on the absolute size of an area in which achange is detected for a specific inspection target structure or therelative size of the area in which the change is detected relative tothe total area of the structure. The calculation of the area may beperformed as follows. First, the ratio between the distance betweenfeature points obtained based on a taken image and the distance betweenthe feature points in an actual structure may be calculated, and anactual area may be calculated by applying the ratio to a change regionwithin the image.

Furthermore, according to another embodiment, an area in which a changeis made per unit time is calculated based on a detected change pattern,and a risk may be detected as being present when a corresponding valueis equal to or larger than a preset value. The area is calculated in thesame manner as described above, and a risk detection signal may begenerated for one of a plurality of levels.

For example, when a change area value per unit time is equal to orlarger than a preset value and is smaller than a first value, a level 1risk is detected as being present. Alternatively, when the change areavalue per unit time is equal to or larger than the first value and issmaller than a second value, a level 2 risk is detected as beingpresent. Then a risk signal suitable for each step may be transmitted tothe administrator terminal 300.

The change prediction unit 240 according to an embodiment serves topredict the change pattern of an inspection target structure in thefuture through a plurality of images acquired by the image acquisitionunit 210 by using a deep learning algorithm.

According to an embodiment, the change prediction unit 240 predicts achange to a structure by using a Long Short-Term Memory (LSTM)algorithm.

An LSTM algorithm is a type of Recurrent Neural Network (RNN). An RNN ismainly used for temporally correlated data, and is an algorithm thatpredicts subsequent data (t+1) by taking into consideration thecorrelation between previous data (t−1) and current data (t).

An RNN has a problem in that it loses its gradient as it goes back intime. The reason for this is similar to the reason for using a nonlinearfunction other than a linear function as an activation function, and isthat a problem arises in that a value becomes smaller as past data iscontinuously multiplied according to an initial value. The LSTMalgorithm was developed to solve this problem, and is an algorithm thatpredicts future data by taking into consideration not only immediatelyprevious data but also macroscopically past data in the process ofgenerating a predicted value.

The table below is a table illustrating the principle of predicting achange to a structure at a later time point through the images X1 to X5taken at time points t1 to t5, respectively.

TABLE 1 X_train Y_train [X1, X2, . . . , X5] [X6] [X1, X2, . . . , X5][X7] . . . . . . [X(t − 5), X(t − 4), . . . , X(t − 1)] [Xt]

Referring to the table above, through the first to fifth images (X1, X2,. . . , X5) acquired at times t1 to t5, the shape of a structure at timet6, i.e., a change pattern, is predicted, and a sixth image may begenerated. In this case, the time intervals between time points t1 to t6may be the same. Also, an image of the structure at time t7 may begenerated based on the images of the structure at time points t2 to t6.

FIG. 5 is a view showing a result image generated by performing changeprediction according to an embodiment of the present invention.

Referring to FIG. 5 , it can be seen that when a plurality of imagesacquired at different time points is input, the image generated bypredicting a change to a structure at a specific time point in thefuture based on the LSTM algorithm is the same as the image of thestructure actually taken at the specific time point.

The change prediction unit 240 may determine whether there is a riskbased on the absolute size of the damaged area of a structure at aspecific time point as a result of the prediction or the relative sizeof the area in which a change is detected relative to the total area ofthe structure. Furthermore, the change prediction unit 240 may calculatethe area in which a change is made per unit time based on a detectedchange pattern and may detect a risk as being present and transmit acorresponding signal or information to the administrator terminal 300when a corresponding value is equal to or larger than a preset value,like the change detection unit 230.

According to an embodiment of the present invention, a change, such asdamage, to an inspection target structure may be accurately detected ina convenient way using a drone.

Furthermore, via a machine learning algorithm, feature values areacquired from images taken at different time points and a change to astructure is detected through comparative analysis, so that accurateanalysis related to damage to the structure may be achieved.

Meanwhile, the prediction of a risk and preparation for a risk may bemade in advance by predicting a future change to a structure based onacquired images.

The foregoing description of the present invention is intended forillustration purposes. It will be understood by those of ordinary skillin the art to which the present invention pertains that each of theembodiments described above may be easily modified into other specificforms without changing the technical spirit or essential features of thepresent invention. Accordingly, it should be understood that theembodiments described above are illustrative but not restrictive in allrespects. For example, each component described as being in a singleform may be implemented in a distributed form, and, likewise, componentsdescribed as being in a distributed form may also be implemented in anintegrated form.

The scope of the present invention is defined by the attached claims,and all variations or modifications derived from the meanings and scopeof the claims and their equivalents should be construed as fallingwithin the scope of the present invention.

What is claimed is:
 1. A method for an image analysis server to detect achange to a structure by using a drone, the method comprising: receivinga first image of the structure taken at a first time and a second imageof the structure taken at a second time by a drone; acquiring a firstfeature map for the first image and a second feature map for the secondimage using a machine learning algorithm; detecting a difference betweena first feature of the first feature map and a second feature of thesecond feature map; and predicting a change to the structure based onthe detected difference using a recurrent neural network, whereindetecting the difference comprises, determining based on an absolutesize of an area of the structure in which the change is detected or arelative size of the area of the structure in which the change isdetected compared to a total area of the structure in the first featuremap and the second feature map, and determining a first risk level whenthe change area value per unit time is greater than or equal to a firstpreset value and less than a preset second value, and determining asecond risk level when the change area value per unit time is greaterthan or equal to the preset second value and less than a third presetvalue, wherein the recurrent neural network comprises a long short-termmemory algorithm generating a new image capable of predicting the changeof the structure over time.
 2. The method of claim 1, wherein the secondtime comprises a plurality of time points, and the second image of thestructure comprises a plurality of images taken corresponding to each ofthe plurality of the time points, wherein the new image is generatedbased on the plurality of images using the long short-term memoryalgorithm.
 3. The method of claim 2, wherein an interval of time betweenthe plurality of the time points is constant.
 4. A system for detectinga change to a structure by using a drone, the system comprising: animage acquisition unit configured to receive a first image of thestructure taken at a first time and a second image of the structuretaken at a second time by a drone; an image learning unit configured toacquire a first feature map for the first image and a second feature mapfor the second image using a machine learning algorithm; a changedetection unit configured to detect a difference between features of thefirst feature map and the second feature map; and a change predictionunit configured to predict a change to the structure based on thedetected difference using a recurrent neural network, wherein detectingthe difference comprises, determining based on an absolute size of anarea of the structure in which the change is detected or a relative sizeof the area of the structure in which the change is detected compared toa total area of the structure in the first feature map and the secondfeature map, and determining a first risk level when the change areavalue per unit time is greater than or equal to a first preset value andless than a preset second value, and determining a second risk levelwhen the change area value per unit time is greater than or equal to thepreset second value and less than a third preset value, wherein therecurrent neural network comprises a long short-term memory algorithmgenerating a new image capable of predicting the change of the structureover time.